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ABSTRACT , * ^ 

Program documentation is an important role for 
evaluation in both formative and summative modes. It should describe 
processes, materials, participants and social interactions which 
produce certain outcomes. Program* evaluation should judge the "effect 
of the program and depict effective practices, products and outcdmes. 
To demonstrate program implementation and achievement, thorough ( 
dpcumentation is required. Documentation methods should suit the- 
needs of the individual program. These needs are likely to be 
influenced by audience; credibility; planning needs; decision makers 
at local; state, or other levels; time factors; and the nature of the 
program. At a minimum, documentation should provide information on 
program frame factors such as participants, resources and * facilities , 
and program events and activities, including materials used and 
procedures. The. principal approaches, used singly or in combination, 

father information directly from participants by written 
escriptions, . interviews, or questionnaires; program records; and 
direct observation of the program setting. Three points of view 
toward documentation validity, are (1) consistency with the program 
plan; (2) plan-free documentation; and (3) ^bheory-based 
documentation. These three points of view are illustrated with 
suggestions for? practice and setting up the documentation system. 
(CM) ' 
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EVALUATION AND DOCUMENTATION: MAKING THEM WORK TOGETHER 
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INTRODUCTION 0 • 

*"t k * .... 

■ * • • « ' 1 

When we evaluate a program, we should keep in mind the major 



L k uatio\v ' 1 " " 



purposes ot that evaljaatiof)^ that is, the role of' the evalua- 
tion in the life of the prograun. Ohe-role of evaluation is fdrma^ 
tive? it serves to help and advise program piannets and developers 
to describe and monitor program activities , 4 assess ■ the progress 
achieved, pick up potential problems, and find out yhat areas need 
improvement. Another major role of evaluation' is sunfmative; it is 
designed to provide a summary statement 1 about ttjfe general ieffec- 
tiveness of the program: to describe it, judge achievement of its 
intended goals, pick* up unanticipated - dut comes , and possibly com- 
p^tre the program with^similar ones. 

Both thfe formative and *?he summative evaluator should be coh- 
cerned with prdgram description, inffch illustration of the dynamics 
of a program: its processes, materials, participants, and social 
interactions which produce certain outcomes. Describing a program 

* * t IS 

in this way points, to a 'third role fpi evaluation, qne that -is 

useful in both formative and summative modes. This is the role of 

f 

program documentation. 

It is important to note that I am' discussing evaluation, 

rather than research. The distinction between evaluation and 

' ' • ■ v ■ * 

research is ah important one, encompassing not only questions of 

/ * 



topic or technique, but- also fiffndamehtal differences in. how infor- 
mation is generated, the kinds -of information generated, and their 
intended uses. Following Cronbach and Sujagfes. ( 1969) who cite the 
need to distinguish* decision-oriented from conclusion-oriented 
investigations , I categorize evaluation and documentation as 
decision-qjriented study -and research as conclusion-oriented study. 
Decision-oriented evaluation provides information .on program re- 
suits relevant for a particular program at a specific time. It^ 
differs from conclusion-Qriented research, not .so much in matters 
o^ topic of interest or techniques employed , ^but rathe* in the 
motivation for cohducting the work, constraints imposed by the in- 
stitiitional setting in which the work is conducted , * and intended 
use of! the information generated.*" Thus, the first role of evalua- 
tion should be to generate information that will lead to decisions 
about educational questions and problems in a particular set- 
ting, while its secondary role is to generate information of use 
in other settings. 

f A " great deal of what is called "evaluation" is in a 
conclusion- rather than a decision-oriented vein. Much of it 
deals with only the investigator's rather than the decision 
maker's information needs. Many evaluations lack in description 
and documentation'? they describe outcomes without describing the 
processes' and .events that led to them: the „ interaction of 
instructional strategies and implementation techniques. Such 
evaluations offer a somewhat stilted picture of the , program 
(Burry, 1979) 



A. good program evaluation, then, should .rfbt only judge the 
effect of the program but also depict its effective (or ineffec- 
tive) practices , products , and outcomes. It should demonstrate 
how the program was implemented as well as what.it achieved; this 
requires a thorough description and documentation Of the program. 

Implicit in the concepts of documentation 4)nd evaluation is 
the desire to discover those effective practices maintained in ttfe 
parent site which may^t^en be adopted at mother sites. Such repli- 
cation or adapt at ion-Vf program practices; depends strongly on the 
evaluator's description of the program's crucial features. These 
features include the activities, materials, pedpia, and social 
factors that constitute the program and, combination, led to a 
set of outcomes . < " . 

There is no singled universally appropriate way,, to gather 
sufficient documentation. The methods chosen should suit the doc- 
umentation needs of the individual program and are- likely to be 
influenced, by: , • • 

* • ' 

\ 1 . The audiences — the people who will examine the results of 
the documentation either as a separate entity^ or as part 
of an evaluation. 

2. The willingness of the audiences to accept*^he documenta- 
tion as credible. 

3, The needs of those who may pian to adapt/ ^dopt thfc program* 
in the future. ' \ . ' \, J 

^4. ,Other decision makers at local; state, regional, and 
federal levels. , . 

The amount of time and help available to do the job. 
6.. the nature of the program documented. 



WHAT IS PROGRAM .DOCUMENTATION? / 

% \ ^ • f P - 

Documentation of a program is aj compilation of records of the 

* ( 

program' s characteristics — participants, materials, activities, 
events, social factors— *and 'a report (a§ part of the pver^ll eval- 
uaJtionj of what the program looked like- .It should describe all 
the crucial features of the program and be considered^ normal 
product of program operation and evaluation. Xfc can include a' 
wide variety of materials such* as attendance records, minutes of 
meetings, internal communications, materials £nd/or their descrip- 
tions, records formally set up to provide sg^cific kinds of docu- 
mentation, participant reports, interviews, and pbservations . . 

tf positive results are found in a program, that program may 
be worth repeating, both to verify its effects andflfco give more 
people a chance to benefit from it.. For these reasons, a simple 
statement of outcome is not sufficient in itself. Also needed is 
a thorough description which, in conjunction with statements of 

outcomes will document the dynamic interactions that helped de- 

^ . • - * T. • 

fine the program and its accomplishments. Similarly, if the re- 
sults were negative, a thorough documentation might help determine 
what went wrong, _ ~ * 

Documentation Should be" detailed enoughs «P serve, as. a source 
for. future planning. If another institution is impressed by a 
program and wants to x implement it; such precise document a£ ion will 
rirovide a source, of planning "for the new participants in their owjv 
setting. 
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Documentation, then, as part of 'evaluation,. . should help in 
data interpretation and "in interrelatibn' of process' and outcome, a 
highly interpretative endeavor. _ r # * 

At a minimum, documentation should provide information on €Me 



t 



following: ' 

t, The fratoe factors of tj?e program: the participants, resources, 
and facilities available. ^ . 

' # * 

2. The events and activities in the program: hdw materials wete 
used and if they were used as ^intended ; r ,antf the procedures and 
,whether they were followed or modified to meet evolvang 
needs. 

» ■ . 

These kinds of activities are time- and people-related events 

that occur during the program. It is more difficult to ascertain 

that proposed program activities are occurring t l han* that a certain^ 

number and type of materials and participants' are present. Also,, 

some prograhi. activities"* are difficult to define and spot.* 'For 

exampS^e, while we may easily document the presence of an aide in 

the classroom, how do we spot the presence of ""reinforcement" or 

"acceptance "of 'cultural diversity" in a classroom setting? 

These intangible entities cannot be documented in the 

straightforwardly that the, presence 1 of materials and numbers of 

participants can be verified. The choice of documentation tech- 

niques is therefore a difficult' and sensitive business. 



How extensive should the documentation be? * 

' lite documentation methods chpsen will depend on the factors 
mentiorffed earlier— the audiences , the time and other resources 
. available , and the nature of the program itself. 



' The Audiences . IJocumentatiori must be responsive both , to in- 
ternal program needs and. to the needs and questions of different 
external audiences. The need for and amount of documentation ma- 
terials collected on program' activities, depends primarily on the 
belief of the audience that the content of the documentation accu- 
rately portrays what happened in the program. For instance, an 
audience -consisting largely of participants in the program would 
probably\be in. a good position to know if the documentation gives 
an accurate picture . * On % the other h^d, external audiences luch 
as funding agencies may be less inclined to accept the documenta- 
tion at face value. If the audience consists of a group of people" 
who may implement the program or parts of it at another site, then 
the conern in documenting will be to comnknjfcaH^~i:fie program's 
most crucial* features to ejnsure that, the documentation describes 
what 'occurred . „ . I 

Time Available , Time determines the extent to which a pro- 
gram can be fully documented. Examination and interpretation of 
primary documents and records, construdt-ion of special * dogumenta- 

tion materials, interviewing, and ^observing are time-consuming. 

v « 

While many, things in a program. can conceivably be documented, it 

Is^wise to narrow dowrt the list of* thes'e docum x entable < activities 

» - * ** ■ 

to those of highest priority r those features that C9ntribute to 

the overall effect of the program. 

1 f v 
The Nature of tfre Program . This factor also determines the 

amount of documentation needed. T^e , amount of ' documentation a 
program demands is often a function^er^tha^program 1 s tangibil- 
ity — how much it relies on observable mater ial^^r^l^tive to spe- 



cif ic < activities. For example,\a program , consisting of a serie# • 

• ' , • 4. 

of printed, published materials is a fairly tangible program, 

while *dne that consists primarily .Of the implementation of prod- 

esses (collaborative program planning in bilingual education H or 

values- clarification in multicultural education) may have to rely 

,heavily on direct observation to document that these intangible 

processes indeed occur. s , . « 

f Further, many programs make strong prescriptions about the 
Mature of activities that are to take* place and how often they, 
should happen. These demand more extensive documentation. A 

teacher training program, for example, where trainees are being 

• » *• 

" instecrefefr to use" reinforcing techniques in a multicultural set- 

**. lit . - * 

ting, will need, extensive documentat fbn,. Direct observations, 
possibly by^Several .observers, of hqw the trainees conduct them- 
selves in ^his^i<uatiol^--in the classroom with children or when 
receiving training .at ^a^eacher training institution— will be 
needed. Observers w^j^i^^-to ( 'be trained and informed about 
exactly what "r2inforci«j^! a«j^vlties look like. ; 

Another si^iafei^n ^i^pPcalia for extensive documentation is 
one in which the^plannei prbgram^Specif ie>s certain minimal levels 
of \ehaviior *(£rdm the participants, for example, 'where' aides or 
volunteers work With student^ St their desks * 50 percent .of the 
day.* .To determine whether this activity:is as pervasive as in- 
^tended, extensive documentation will be needed. Knowing how often 
'and- what percentage of /• time such activities have occurred will 
lead "-to better -interpretation of process -and outcome in the pro-r^ 
gram. 




4* ' - ' 
A documentation effbrjt will not always rely, exclusively , on * 

the Irecords or documents/ that naturally evolve during the course 

of the program. Depending on the nature of^tjfte program, we may 

want to influence the kind\ of , documents that evolve ; so that they 

will meet. our patticular documentation <needs* «^ 

«. - • \ ' / 

I DOCUMENTATION OP FRAME FACTORS 

^ ■ ■ ■ > \ * 

This documentation will look primarily at the participants 

. \ , ^ * 

and materials that are part ^of a, program. 

' " \ ' f . 

Some questions related to t^is part of the documentation* are: * 

0? • • •• ■ • ■ ' ^ " " 

irs-cfrinel 



Per 



What percentage of parents participated? 



Are these parents typical of the fcffplV schobl community or 
only of certain groups? . v . . * 

What special characteristics, training, abilities, or 
•community affiliations are required for staff members? 

Did the staff meet these requirements? 



Materials 



What new instructional materials were developed? - 

How did this development occur? By whom? When? «' 

Where can a set of these materials be found f Or* examination? 



Participants 



a i 



Who are the students in the program? 

How old are they?. , '•' tffa 

Whales their linguistic background? s ,. 

What are their SES levels?) 



How were students/classes selected? 
. How do they compare to other students in the district? 

'•' . OOCUMENl^TION OP PROGRAM ACTIVITIES , . 

, . * . .i 

This documentation will,, look primarily at the* events and 
activities of the program. People who may follow the program will 
need to know exactly what processes and activities occurred.* If 
people want to form^easonable hypotheses about what critical pro- 
graal "factors helped produce 'its outcomes, documentation will pro- 
vide needed information. Further, discoveries of this kind will 
help form future policy decisions, thus giving evaluation a wider 
universe of relevance than would the single-project focus. 

Where a project intends to implement a particular program and 
produce certain .outcomes, documentation can demonstrate its ac- 
countability . . This is particularly true in cases where a program 
aims toward outcomes far distant in the future or, those that are 
difficult to recognize or measure accurately. In these* cases, 
documentation . may be the only basis for short-term 
accountability. 



.THREE POSSIBLE METHODS IN POCUMENTATION* 

There are three principal approaches to documentation that 
can.be used -singly or in combination: 

1. Gathering information directly from prpgram participants: 
students, teachers, aides, parents, administrators, etc. 
These participants are asked to write descriptions of 
program activities. As an alternative , they may be in- 
terviewed or asked to complete a questionnaire. 



•f * 

2. Using the records kept over, the course of ' the program (as 
' well as influencing the generation of Icinds of. particular 

records that would be use'ful to specific documentation 
needs) . 

* d 

* W> 

3. Conducting direct .observations of the program operating 
in the classroom or elsewhere. 

. - V " • . 

* . * ■ * ! '. ' 

How do you make decisipns about what to use ? 

What sort of documentation information would be most useful 
in a Situation? Vfofrld the evaluator like to know ^ow well the 

implemented program fit the original proposal? t Or is the eval- 

* ' ' * 

viator primarily interested in program outcome , regardless of the 

proposal?' Does the evaluator want information on the program yith 

respect to a particular model, philosophy, or theory of education? 

That is, does the program represent an example of this model, 

philosophy, or theory of education? These questions - highlight 

three different points of view that might be' taken toward program 

documentation. Each "poirtt of view has a slightly different 

•v * 
purpose, but the purposes are not mutually exclusive. 

1. Document.ation of consistency with the program plan. Doc- 
umentation from this point of view vjill involve collect- 
ing data to determine the extent to which the (crucial) 
activities as planned did occur; and if they did not oc- 
cur, what happened instead. , 

2. Plan-free documentation. Sometimes program ijnplementors 
lack consensus about what ^constitutes a program's crucial 
activities;' sometimes these activities cannot be identi- 
fied until the program begins. In this situation, and 

• * 4 when plans are changed to meet evolving needs, plan-free 
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documentation will be useful. What actually occurred 
will be, documented, regardless of what was intended. * v '"' ^ 

• » A / 

i 

Documenting from the plan-free point of view, however, 
does risk omission from the documentation of various 
activities that some participants Vmight feel were 
crucial. Its advantage is t!hat a relatively unbiased 
picture of what the^ program actually^ looked like in 
operation will occur , and perhaps pervasive and crucial 
activities not originally conceived will be uncovered. 
3. Theory-based documentation. This approach permits pro- 
gram - documentation* in terms of consistency with a theo- 
retical prescription of how these activities should look; 
that is, whether the program is consistent with a speci- 
fied educational model, theoiry, or philosophy. 
Documentation, in terms of theory, will provicfe valuable 
information for £&feui|e planning, especially if -the docu- 
mentation is tied into evaluation measures of outcome". 
If the program has outcomes that are difficult to measure 
or that will occur far in the future, at least the docu- 
mentation will demonstrate accountability for implement- 
ing activities that, by virtue of model or theory, are 

i . ■ 

likely to r bring about these outcomes. " 
While one particular documentation point of view may assume 
primary importance, the overall plan can call for the use of more 

than one viewpoint, if appropriate. In this way, a documentation 

« 

effort might describe achievement of intentions (consistency) , 
achievement of unexpected or unplanned effects ( (Plan- 



ree), and the contribution of a particular educational philosophy 
(theory-based) . 

INFORMATION GATHERED DIRECTLY FROM PROGRAM PARTICIPANTS 

Although the credibility of findings based on participant's 
reporting may be challenged by some, they can still contribute to 
a .fairly 'accurate , description of program activities. Gathering 
participants' reports will involve questioning participants'* ^espe- 
cially in terms of, but not * necessarily limited to, consistei^y 
4*ith program plan or 'theory. There are three ways of accomplish- 
ing documentation via staff reports: 

1.. Participants can write a report of what they did and how 
these activities are consistent with those prescribed. 

2. Questionnaires can be constructed for participants. This 
will involve .forming a l,i,st of critical activities and 

framing, questions about their occurrence. 

* * ■ *■ ■ . * 

3. Participants can be interviewed. The interview can ei- 
ther be structured (demanding the predesign of an in- 
terview questionnaire) or unstructured (requiring 
that participants describe how what they did was consis- 
tent with the program prescription). * In the Matter meth- 
od, extensive notes must be recorded and transcribed. 

a '■} 

Participant Reports 

Eliciting participant reports is advantageous in that a spe- 
cial instrument can be quickly designed for them. An vadded ad- 
vantage^ is its focus on participant perception ^f the program's 
importrsrtit activities. These reports can be of two jcinds: period- 



ic reports throughout the course of the program or retrospective 
reports at its end. 

Periodic reports wi^l yield fairly accurate documentation. 
They do not make demands on memory like those required by end-of- 
program reports. On the other hand/ end-of -program reports / "and 
the resultant introspection/ may provide a picture more tempered 
by reflection. 

Questionnaires 

Questionnaires mu^t be written to elicit responses that are 
accurate descriptions ti\e progo£am| There are two formats com- 
monly used in writing questionnaires— closed (elected) or ^open 
(constructed) responses. Closed response scores easily and re- 
ports clearly, but tends to lock respondents into answers dicated 
by* the items. Open response is time-consuming/ but it allows re- 
spondents to 1 express themselves freely and may provide a fuller 
picture. A. questionnaire can — -and frequently should — consist of 

both kinds of responses. 

. i ■ j 

Interviews .„ r * 



Using interviews for program documentation Jias several advan- 
tages over participant reports and questionnaires. The* ii>terview 
is a personals face-to-face contact. This offers flexibility in 
asking questions and provides an opportunity to obtain complete 
responses. If f for instance/ it is discovered that a particular 
activity has not occurred/ interviewing allows this question to be 
pursued further/ probing for the respondent's opinion about^why it 
was not used and di^overing whether it was used in ^tjjer class-. 



rooms. Such questions can provide more complete information than 
might the questionnaire or the participant report; 

Another advantage of the interview is that the information 
gained will be more complete. Participant reports must rely upon 
the writers recall of what occurred, and the "writer's memory 
may be /'faulty. In the interview, the respondent can be cued and 
questioned 'about specific areas that might otherwise be over-, 
looked. If carefully handled, the interview offers maximum access 
to information It cah, however, be time-consuming . 

« The form of ti*e interview will depend on/the frame of refer- 
ence chosen for asking questions. Like the cldsed. response ques- 
tionnaires, an interview can be structured, baskd on specific ques- 
tions in a prearranged order. Neither the questions nor ^ their, 
order are varied across interviewees. It can also be unstruc- 
tured, consisting of a few general, questions with predetermined 
probes designed to allow the respondent to amplify answers. 

Self-report information from the ' program participants can be 
adequately co\rered through staff reports , questionnaires , and 
interviews. Yet, reports, questionnaires,, and -interviews of staff 
members, no matter how well carried out, rely upon the opinions of 
the staff pembers and the information they are willing to give. 
Questionnaires and interviews, further, may be poorly constructed, 
leading to biased information. In such cases, credibility can be 
increased by examining records or by direct observation. 
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EXAMINATION OF RECORDS 

Records can provide systematic, tangible accounts, of prografc 
occurrences. There* are two things to c6nsider about program docqf- 
'mentatp.on by record examination:^ ( 1 ) using existing program re<J- 
ords, and (2)* designing special recordkeeping systems for the doc- 
umentation. -„ 

• \ 
It is generally possible to use existing records kept over 

the course of a program already underway. In the absence of rec- 
ord systems specifically designed for documentation purposes , 
records normally kept for program administration or materials that 
result ifrrom the implementation* of program activities will often 
yield a considerable amount of the information required. / 



An important advantage to using records is that they provide 

more 



f credible information than staff reports.^ They also provide 



information on program activities without troubling the partici- i 

. - * w 

pants with requests for written - reports or with time-conrsuming 

questionnaires and interviews s Records kept for the program^ how- 
ever, may not contain all the information requited for v an ' accurate 
program description. This is why records are more commonly used 1 
as sources of documentation • for only some program activities • ' 
Unless recordkeeping is carefully planned for particular . documen- 
tation needs, records seldom contain the whole story, and what 
they do contain may be inaccurate. j ' : ' : ! ' 

Another factor to consider is that, aft times, there may be 
legal or ethical barriers to record examination. Problems over / 
the legality of recoil examination often can be overcome by assur- 



ances that data collected will be presented in summary form and 
will keep -confidential the identities of individuals . Docunfenta- 
tion by means of_r^cord examination s can also be a time-consuming 
process: It takes time to obtain records, extract relevant infor- 

\ 

mation from them, and summarize the data (Stenhouse, 1.977). 

OBSERVATIONS * * 

Any documentation can' prof it from on-site observation of the 
program in action. In many cases, thi^ will merely involve the 
observer's presence during a program session. ^In other situa- 
tions , . observations of the program in' action will be a major 
source of documentation data. Some people -feel that observation 
is the method for performing program documentation. Observa- 
^tion, however, may itself interfere with the working o£ the pro- 
gram!, cursing anxiety and affecting the routine operation of the 
very activities to.be observed. \ 

Observation techniques can be informal or formal. Informal^ 
observations involve a casual look at the ^program by a few ob- 
servers, followed by preparation of a report. They are quick and 
easy to achieve. Formal (or systematic) observations 

follow ciltaore strictly controlled 'methodology , specifying observa- 

n >W. J !,'-A>'. 

tion times and recording methods. 

Systematic observations shotild be considered when: 

1. The program is guided by a highly specific p^lan or theory. 
The stronger the prescription , the more systematic the docu-* 
mentation should be. - 

2. TJhe 'documentation and evaluation may be used to make generali- 
zations about the causes of the program's outcomes. 
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3. The, Program ' s audiences may ask &or strong supporting, data. 

In the absence of these kinds of conditions, systematic obser- 
vation may not be necessary; but these, conditions are likely to be 
there in some form or other. % 

The credibility attributed .to systematic observati^h methods 
stems 'from the^clear definition of 'these techniques; they £>re- 
scribe what observers- will watch - and how they will record their * 

t »> S 

observations . V • 

The purpose of systematic observation is to provide evidence 
of whether a, program wa^ implemented as planned. But the observ- 
ers do not judge;- rather, they are human recording instruments who 
are transferring" to paper a record of 'the activities that occur as 
the program operates . * - 

Depending on the point of view from which * the evaluation is 

being undertaken (plan-free/ plan-based, theory-based, or some 

* v » 
..combination), obse««ers may or may not be preinstructed about what 

behaviors to look for. 

We can oxqanize observation methods according to; 

V . 

1. Whether o\ not the behaviors to be observed ^ave been prede- 
fined. % N - 

2. How observers have been instructed to behave during and after 
the observation session — Whether they sit and watch, write ,v 
make checks on a checklist)! or reconstruct what, happened #fter 
the-^observation period has ended. 

Systematic observation is a technique where events and behav- 
iors to be watched (or watched for) can be predefined. JThere are 
three methods available 1 for conducting observations based on|a^r' 
predefined notion of how the program should look: 



iors, waits *for them v to occur. When they do, 
server check^ that they have occurred at least 



ce or 



1. Checklists. The observer, with a list of defined! behav- 

the" ob- 

f " 

tallies their frequency. 4 ) % . 

V ^ . ) I 

2 . Coded behavior reports . . The observer learns codes or 
symbols representing events and behaviors to be Watched , 

, keeps a running record of the sequence of occurrence of 
* .these events, and ignores events for which no symbols are 
provided. s * j ' 

1. Delayed peports. ' Th<e observer is told which events/ be- 
haviors to watch for and then observes the • program in 
operation. When the prescribed period for the dBservte^ 
tion has ended, the observer fills out a checklist or 
questionnaire that ^records the occurrence and form of 
the behavior's that were watfched, asks about the degree of 
frequency with which they occurred (rating scales),- or 
uses a combination of recording methods. „ 
These three observation techniques will be useful for obtain- 
ing slightly different types of data where different preconditions 
exist. Which observation mpthod will most' closely fit the docu- 
mentation needs N will depend on the information required. For 
instance, if frequency of occurrence is the primary concern, the 
checklist may be sufficient. If, however, the typiial* sequences 
of events must be described, the coded behavior wilL/be needed. 

Checklists will be" useful wh£n the* evaluatoir wants to know 
whether otf^liow often certain events or behaviors occurred. They 
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will not he!bpt=iocuinent the form that^h^y tooK, Where many behav- 
iorgT/events are to be watched, it; w^ill be difficult to. use a 
checklist. One . of the two other methods — coded behavior or de- 
layed report — will be more appropriate. A checklist can be used 
to tally frequency or duration of N activities . The method for tal-^ 
'l^n^ frequency differs slightly .from th^t for tallying duration. 
Duration demands some sort of * time sampling, where observations 
are paced by the passing of shoi^t time interval's . Frequency tal- 
laying ~ means tallying each occurrence of a discrete behavior or 
event. 

. Codecf behaVior records ajlow the evaluator to record the oc- 
cur'rence of several behaviors, though each of these behaviors must 
be precisely specified. Essential to the coded behavior record 
method is the assignment of symbols, or codes,' 'to the behaviors 
observed. These codes will be us^d " by observers to produce a 
quick, running account of the behaviors they- observe. To achieve 
the desired account, observers will have %to be trained to distih- 
guish one behavior from another and to learn the symbol system - . A 
good rule-of -thumb is to set up a code system comprising as few 
symbols as possible in order to ensure that observers will neither 
forget nor cortfuse them. If symbols can be combined to provide 
precise descriptions behaviors , the repertoire of possible be- 
havior descriptions will increase dramatically ^depending on the 

number of permutations of combinable symbols). In other- words, 

/ V 
the number of symbols u£ed 'should be as small as possible, but 

theynshould be combinable so that the number of possible precise 

behavior descriptions that can be made will still be extensive. 
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o Coded behavior records are generally more time-consuming than 
simple checklists . Observers must be trained and tested for 
agreement dn the meaning, of symbols they will use to keep track of 
what they observe. Once these observations are taken , data must 
still be extracted from the observation records. Counting tallies 
or scoring checklists take minimal bdme, but coded behavior rec- 
ords produce strings of symbols . These symbol strings must be 
subjected to further interpretation for the information obtained 
to be .suitable far reporting. 

When should coded behavior records be collected? The coded 
** i ■ '. 

behavior record will give mote information about the form of 

behaviors — how they looked dn operation — than will the checklist. 

Depending on how the categories are defined for coding, the evalu- 

ator will be able to detect whether activities happened in the way 

prescribe^ by the program plan or theory. Also', if the activities 

did not coincide with the plan, the evaluator; will be able to de- 

scribe what happened instead. The coded behavior record will also 

provide information about the form the activities took, showing 

patterns of events, activities^, and behaviors. 

' ? 

Two common methods, tracking by event/behavior and tracking 
by time periods, are in general use for setting up a coded behav- 
ior report. When tracking by event, the observer marks a symbol 
each time it occurs, no matter how long it /lasts. When tracking 
by a time period, the observer notes each tim$ a prescribed time, 



period passes. If the individual beirig observed performs an ac- 
tivity for a long time, the observer notes a symbol for the behav- 
ior displayed each time a period of particular length passes. 
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Tracking behavior is njost useful for noting behavior *Dr events 
t^at change of ten. over the course of an observation, or when'the 
e\&fent of interest has a clear* beginning and end. 'Tracking by time 
periods works better where behaviors/events are prolonged/ . Obvi- 
ously, ij: is possible to combine the two methods. %' * 

• <• * ' ' ' - * ■ ' * * • 

^Delayed reports are checklists or questionnaires completed by 

the observer after a ^period of observation .has ended. >° This 

< * X > • . * 

"time^ unit" method, highly systematic, follows a clearly defined 

schedule. With questionnaire or checklist in hand, the observer 

watches short (a few minutes) segments of a participant's be- 

.* * 9 • 

havior, takes f a predetermined number of minutes to fill out a long 
checklist or questionnaire, observes another time unit of the same 
^or another person 1 Ss. behavior , then records, observes^ etc. ° The 
observer can either record notes in the place of occurrence or 
withdraw to record the data, as when such a^ftivi^y would district 
or prqduce., snxiety^iji ,the participants ./ Observations made t^iis 
systematically, panticxiiarly where reliability can &e demonstrated 
between two or mdre observers, are highly credible.' Of course, 
this highly) regulated approach demands that the behamors observed 
be describable via checklist or questionnaire; it also demands . 
that the predetermined time segments be selected from periods of 
the day when the behaviors in question- are likely to occur. 

Delayed reports are most useful where behaviors must be watched So 

x 

carefully during Jfclie observation peViod that recording on-the-spot 
^ would interfere. It is also the method to ehoose when data from 
many participants must be gathered during an observation period.^ 
Whsn the behaviors of many people are involved, observers can plan 
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to watch different people oCffing each time segment: After* record- 
ing behaviors* of , one individual fo^ a given time segment, each ob- 
server can swit.$3i £o a different person* 



The delayed report methodology is less time-consuming than 
setting up and filling out a coding sjysjt^pi. In reality* using the 

delayed report method is not much morte complex than the writing ^of 

- * ,f- *' '■ * "'■ ■ «■ ' 

*a questionnaire. There aire* however, , two procedures that must- be 

* ... # 

followed in addition to developing a questionnaire; 

x1. The 1 observers must be preinformed about what they are 
looking for by having them learn' the questions on the 
1 ■ *. s questionnaire beforeh&pd. \ 

2. The observation session must be formalized ai f much as 
possible by controlling when it will occur, how long it 
will last, and who will be observed. . & 

I suggest th^s close ^emphasis on behdviors not because I 




question an observers 1 ability to "make inferences (though to an 

extent I do), but because inference in a recording irfstrument will , 

-j * ^ "^^^ * 

inevitable affect the^reliability of the observation method, ' Rftf 

liability is the degree t9 which one observer using an observation* 
scheme g6ts the same results as another observer using the same 
scheme. It is a measure of the stability of a measuring^aethod. 
Frequently, reports of an observation method are acc6mpani*d by an 
index •.of the instrument's reliability, i. The correlaticna be- 
tween observations from each of "two observers can -ierv^e as an in- 
dex of reliability. Two or more ■ observers can be. relied upon to 
see the same thing as they watch the same behaviors, but they 
might not i^ake the same inference from what they have seen. 
For the sake of agreement among observers, the inevitable inf er- ^ 
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ences they make must be kept as close as possible to the behaviors 
observed. 

Where inference is inescapable reliability can be increased 
by pretralning observers so that they agree ' about* exactly which 
behaviors s are referred to by each item ort the Recording instru- 
raent. A relatively high^inf erence report system that can demon- 
strate high reliability (a correlation of .80 or. better) will* be 
as- useful and credible as. any other. 

As rioted earlier , knowing which documentation technique^ s) to 
use is difficult. In addition; deciding whether the documentation 
should stress program and plan consi3tency, be plan-free, illus- 
trate theory, or take the form of some pombiiiation is also diffi- 
cult. Further, some of the particular techniques described re-^ 
quire more time and effort in ttveir proper uses. 

Careful documentation, however, adds to the evaluation compo- 
nent an element that fully describes program implementation and 
practices and those events and processes that interacted \o 
achieve outcomes. This goes far beyond the typical process of 
statement of outcome and estimation 1 of effect. It recognized pro- 
gram uniqueness and documents what a particular program did to 
achieve its particular outcomes with a certain kind of student, 

SUGGESTIONS FOR PRACTICE 

What to Look At 

Evaluators, must become thoroughly familiar with a wide vari- 
ety of program feature's. They must buil^ into their information- 
gathering plan a„ variety of formative tasks to include examination 
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of program implementation , using resultant information to suggest 
areas of- program improvement and careful documentation of the 
implemented features . and their relationships. Evaluators must 
also improve their summ^tiye evaluation activities by estimating ' 
the contribution of specific program features to program outcomes. 
Evaluators must single out appropriate program features for 
examination in an evaluation that is both technically sound 
and sensitive to the 1 needs ? ajnd constraints 6f the program 
under examination. 

* These aims can be facilitated by examining the program plan 
or description; determining (in consultation with program partici- 
pants) which program features need to be evaluated; deciding upon 
the means to evaluate these features; selecting methods to ensure 
ongoing documentation of their implementation^ relationships , and 
cumulative effects? and considering appropriate means of reporting 
information to various audiences. > 

• Features most needing evaluation and documentation will N vary 
from progtam to program. Election of these features and manage- 
ment of their evaluation/documentation canvb^ facilitated through 
examining regulations governing the program, program plans and 
records, previously gathered evaluative information , and discus- 
sions with staff. ^Evaluative feedback r on these features may en- 
hance the information's utility for program staff. Such 
reporting, also, will be invaluable to others who may use the 
information to develop and implement a similar program. 



25 



From Which Point of View; Consistency, Plan-Free, B asis in 
Theory, or Combinatio'nT — 

Consistency with program plan . This 1 point of view presup- 
poses the existence of some sort of 'guiding plan or description. 
Collection of documentation data will determine the extent to 
which the crucial activities of the program occurred as planned; 
if they did not, - the documentation should pick up what happened 
instead. This information, together with other evaluation data, 
is most useful to program staff who want formative evaluation 
..information— information that will help them decide whether the- 
^program' is working as intended; if not, why not; areas-where the 
program can be modified and improved; and areas in which the 
program's operation is optimal bu ( t for which a written 
plan was poor, or did not anticipate unforeseen events. This 
approach to documentation is also extremely useful in the 
suapative evaluation of a program; it provides present program 
operators and potential second-generation program users with a 
careful description of implementation and the setting most 
conducive to its success. 

Plan-free documentation . This approach to documentation 
avoids preconceptions about how program activities should 'look. 
It J may be used in conjunction with the consistency approach. 
There - are several situations ill which plan-free documentation 
might be useful:.. ' 

1 .The program has no plan, nor is it based on a theory or 
model; it may be very appropriate in a program that is in 
its very early and. evolving stages. , * - 

* * - 

*' 2. There is a plan, but it is" either too vague or too gran- 
diose to be practical for identifying documentation 
. needs. 
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3. There is an apparent need for more naturalistic documen- 
tation that might be used ,in conjunction with a plan- 
consistency or theory approach to provide a fuller pic- 
ture of the program. 

Plan-free documentation, especially of complex programs; al- 
lows for uncovering important features or activities as imple- 
mented that might have gone unmentioned in the plan. This will 
often occur where there are strong social and political forces 
likely to affect the program. But this kind of documentation, 
as a sole source, may overlook activities crucial to program 
planners. The decision must be made Hhether the trade-off is 
likely to be favorable for producing a useful and credible docu- 
mentation.' r 

Theory-based documentation . This approach is useful for pro- 
grams that attempt to implement a model of teaching, a theory of 
learning, or a philosophy of schooling. These phenomena may have 
identifying characteristics, prescriptions about the nature of 
activities in which the participants should engage. Proponents of 
each theory, philosophy, or model will contend that its particular 
activities will be the cause of the program's hoped-for outcomes. 
Documentation based on theory will then provide a check on whether 
these prescribed activities are indeed occurring and the extent to 
which they typify what happens to participants during program 
operation. This approach is also particularly appropriate for 
outcomes that are not directly measurable or that will not be 
realized until some futuref* time, often after the program has 

e 

ended. In this kind of situation, it is best used together :with 
one of the two methods described above. 
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Setting Up the Documentation v 

The following suggestions are intended to help efl;s«re a docu- 



mentation system that will provide v&lid, reliable, and useful in- 
formation. They amplify the three basic approaches to documenta- 
tion outlined earlier. 

1,. Information gained from program participants f t iffis can 
consist of reports , questionnaires , and interviews . 
a. Participant reports can be periodic and gathered 
throughout the course of the program, or retrospec- 
<& tive and collected at its end. . 

• 'If — se^y ' on periodic reports, inform partici- 
pants about the documentation effort, why you need 
this information, and hpw it will be useful to -pro- 
gram participants as well as to the evaluation ef- 
fort. List the reports you will need. List the 
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participants involved and the dates on which you 
need reports? prepare instructions for the writers, 
and gather and interpret their reports. 
•If you rely on retrospective reports, decide when 
you will need them, and provide writers with suffi- 
cient lead time to meet your deadlines. Decide how 
many reports^ you will need, keeping in mind con- 
cerns with data reliability and credibility. Pre- 
pare instructions for the writers and gather and 
interpret their reports. 
Questionnaires can be closed- or open-response 
format. The steps in questionnaire development con- 
sist of: * „ - 



• Preparing a program activities checklist. This 
checklist will refer to the program plan or theory , 
level of detail, frequency and duration of events, 
/form that the events take, and degree of 
participant involvement. 

• Considering question categories. Consider accuracy 
of questions in terms of occurrence, frequency, 
duration of activities as well as . form and degree 
of involvement. 

• Designing the questionnaire. This can take the 
form of closed response, open response, or a combi- 
nation of the two.. 

• Critical- examination, of the questions: Are they 
straightforward? Do they imply a hidden agenda? 
Will they lead to biased answers? 

• Testing and revising the questionnaire as neces- 
sary. Are the questions interpreted in the way you 
intended? 

• Administering, analyzing, and interpreting re- 
sults. 0 

Interviews can be structured or unstructured. , In 
writing questions for the # structured interview: 

• Prepare a program- activities checklist (with the 

h 

same kind of concerns followed for questionnaire 
^development) . % 
•Generate questions based on the list of activities; 
prepare alternate phrasing and probings to meet 
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riohexplicit responses. Again, the focus is on 

.1 " •' * 

occurrence, frequency ,' duration , form, etc. 

•Critically examine the ' questions make sure the 

0 

questions do not answer themselves. 
In writing questions for the unstructured interview: ^ 

• Follow the same focus as above, but generating only 
a few general questions to be followed by probes to 
elicit detail. 

For both forms of the interview: 

• Assemble the interview form: Make sure questions 
are in logical order and that there is plenty of 
space to write answers. ' 

• Write an introduction that sets the tone, state's 
, why you need the information, how you will use it, 

and describes data" confidentiality . . 

Si-'' ■ ' 

• Rehearse the interview, looking for inconsistency 
in the' logic of question sequencing and, difficult 
or threateningly worded questions. 

Examination of records . These can be record systems de- 
signed specifically for the program's documentation or 
they can consist of records' that naturally evolve during 
the life of the program. Again, set up a program activi- 
ties list reflecting plan, theory, or philosophy. Find 
out which program records will naturally be maintained. 
Determine /which of these records will meet your documen- 
tation needs in terms of » program 'events and their- occur- 
rence, duration, form, and participant involvement. 
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a. if t£e planntfd^records do not entirely meet your doc- 
umentation needs, discuss with staff/participants 
ways to incorporate the records you want* 

b. Prepare a plan for collecting and extracting data: 
„ You will probably no)t be able to collect and tran- 
scribe data- onVvery participant and event over the 
course of the program. What's more, to pro^u^e a 
detailed and credible documentation, you may not need 
to cover the, entire spectrum of activities. 

There are two possible record collection plans for reduc- 
ing' the .amount of data: 

a. Reduce the number of records to examine by selecting 
v. a sample (random, representative) of participants 

and events and base your documentation on these 
records, only. * 

b. Reduce* the number of records to examine by sampling 
time units over the course of the program: sam- 
ple of days, weeks, or even months. 

Observations . These can be informal or systematic. If 
you. rely on informal observations: 

a. Explain to the observers how the pfogram should look 
or describe its underlying theory or philosophy. 

b. ' Tell the observers what to^look for. 
Have each observer ^.ook at the program two or three 
times during its operation. 
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dV Ave observers write reports describing what they , 
saw, or give^each observer a questionnaire, or inter- 
view them. 

Systematic observations can consist of checklists, cod^d 

J- ' , ■ 

reports, or delayed reports. i 

Systematic observation consists of placing one or more 

observers into a real situation in the program. The ob- 

■^servation is systematic because:' 



a. Observers enter at a preselected, typical tiflre in the 
program! s*. life. * 

b. They stay for a pre-appointed length of time. 

c. They are usually guided in their observation by some 
form of pre-ins&ruction. * 

The steps involved in carrying out systematic observa- 
tions»consist of the following: 

a. Construct an activities checklist reflect plan, the- 
ory, and philosophy'. * 

b. Observe the ptogram in operation to get a perspec- 

t- 
ive about what the observers are likely to encoun- 

ter; this will help you with questions for the obser- 
vation instrument. 

c. Prepare an activities scenario tQ cover the events 
and behaviors ' that should take pl*ce as well as 
those that should not. It is important to get 
information on both. 

d. Choose an observation method: checklist, coded 
behavior report, or delayed report. The method 
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chosen depends on the events you described in your 
• scenarios as well as your documentation needs. 

Checklist \ 
• *• Use a checklist when: 

1. You want to measure how long a "behavior lasted or how often it 
occurred. 

2. You want to measure participant involvement and you, feel that 
duration or frequency of behaviors* of 'the participants will 
reflect involvement. 

3* Your scenario depicts (or you wish to document) very few (six 
or fewer) crucial behaviors to be monitored. ■ 4 

*4. The scenario depicts relatively few behaviors to be displayed 
in a fixed sequence. • w 




Coded Behavior Repoi 

Use a coded behavior ^report when: ^ 

1. You want to record the sequences in which behaviors occurred , 
including those sequences not prescribed. 

2. You must document many (more* than 6 >and as many as 100) 
behaviors. ■ * 

3. The amount of time available ^apd your own expertise make it 
t feasible to devise a code for recording {behaviors. 

4. You want to record as much as possible of what the observer 
sees. ^ - * . * . 



Delayed Reports A 1 < 

Ose r the delayed report method when: 

1. You f ek you can get better data by sampling an individual f s 
behaviors* periodically , rather than recording whole episodes; * 
for example, when yo/ need data' on many people working 
independently. 

2. You are relatively short on time for training observers and* 
processing data. 
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3. Behaviors must be watched so carefully that recording would 
interfere. • 

4. There is a possibility that recording would intimidate , 
fascinate, or otherwise disturb the people observed. 

- • t 

Regardless of which observation technique is selected , you 

0 

must also decide how long each observation should last in order to 
get a good picture of the program.. You will also have to decide 
who to* observe and when, over the course of ttve program, 
to observe. It is better not to do all the observations over a 
limited time period, but rather to space them put over the pro- 
gram's beginning; middle, and end. Spreading out the observations, 
in this way helps to detect changes in the program, increases rep- 
resentativeness of the documentation, and therefore helps increase 
its credibility as well as its interpretive power for the evalua- 
tion. « 

To thl . extent that it is necessary, observers will heed to 
be trained in the use of the/^ particular technique and the 
phenomena to be "observed. This training is critical to the 

la 

' qeneraticfh of valid and reliable documentation information. 

Perhaps the best way to get a full and accurate picture of 
the documented program is tp use multiple measures arid data gath- 
ering methods that let you, assemble converging data. To the ex- 
tent that it, is appropriate to the program arid feasible, a" 

^"combination of interviews, records, and observations can be used 
to generate information that supports or qualifies the picture of 
the program gai*n£d by each single approach.. 
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